Assessing the depth of cognitive processing as the basis for potential user-state adaptation

Objective: Decoding neurocognitive processes on a single-trial basis with Brain-Computer Interface (BCI) techniques can reveal the user's internal interpretation of the current situation. Such information can potentially be exploited to make devices and interfaces more user aware. In this line of research, we took a further step by studying neural correlates of different levels of cognitive processes and developing a method that allows to quantify how deeply presented information is processed in the brain. Methods/Approach: Seventeen participants took part in an EEG study in which we evaluated different levels of cognitive processing (no processing, shallow, and deep processing) within three distinct domains (memory, language, and visual imagination). Our investigations showed gradual differences in the amplitudes of event-related potentials (ERPs) and in the extend and duration of event-related desynchronization (ERD) which both correlate with task difficulty. We performed multi-modal classification to map the measured correlates of neurocognitive processing to the corresponding level of processing. Results: Successful classification of the neural components was achieved, which reflects the level of cognitive processing performed by the participants. The results show performances above chance level for each participant and a mean performance of 70–90% for all conditions and classification pairs. Significance: The successful estimation of the level of cognition on a single-trial basis supports the feasibility of user-state adaptation based on ongoing neural activity. There is a variety of potential use cases such as: a user-friendly adaptive design of an interface or the development of assistance systems in safety critical workplaces.DFG, 325093850, Open Access Publizieren 2017 - 2018 / Technische Universität Berli

Assessing the depth of cognitive processing as the basis for potential user-state adaptation - Data set

EEG and behavioral data of seventeen participants recorded by members of the Neurotechnology Group at Technische Universität Berlin. Details of the study are published in "Nicolae I-E, Acqualagna L and Blankertz B. (2017). Assessing the Depth of Cognitive Processing as the Basis for Potential User-State Adaptation. Front. Neurosci. 11:548, 2017a. doi: https://doi.org/10.3389/fnins.2017.00548EC/FP7/611570/EU/MindSee Symbiotic Mind Computer Interaction for Information SeekingBMBF, 01GQ0850, Verbundprojekt: Bernstein Fokus Neurotechnologie - Nichtinvasive; Neurotechnologie für Mensch-Maschine InteraktionPOSDRU/159/1.5/S/134398/Knowledg

Enhanced Classification Methods for the Depth of Cognitive Processing Depicted in Neural Signals

Analyzing brain states is a difficult problem due to high variability between subjects and trials, therefore improved techniques are requested to be developed for a better discrimination between the neural components. This paper investigates multiple enhanced classification methods for neurological feature selection and discrimination of the depth of cognitive processing. The aim is to detect the strengths and weaknesses of different classification methods and benefit from their highest performances, so that the neural information could optimally be detected. As a result, we obtained a classification rate improved by at least 5% by integrating complementary information that better describe the neural activity.EC/FP7/611570/EU/Symbiotic Mind Computer Interaction for Information Seeking/MindSeeBMBF, 01GQ0850, Verbundprojekt: Bernstein Fokus Neurotechnologie - Nichtinvasive Neurotechnologie für Mensch-Maschine Interaktion - Teilprojekte A1, A3, A4, B4, W3, Zentru

EEG-based classification of video quality perception using steady state visual evoked potentials (SSVEPs)

Objective. Recent studies exploit the neural signal recorded via electroencephalography (EEG) to get a more objective measurement of perceived video quality. Most of these studies capitalize on the event-related potential component P3. We follow an alternative approach to the measurement problem investigating steady state visual evoked potentials (SSVEPs) as EEG correlates of quality changes. Unlike the P3, SSVEPs are directly linked to the sensory processing of the stimuli and do not require long experimental sessions to get a sufficient signal-to-noise ratio. Furthermore, we investigate the correlation of the EEG-based measures with the outcome of the standard behavioral assessment. Approach. As stimulus material, we used six gray-level natural images in six levels of degradation that were created by coding the images with the HM10.0 test model of the high efficiency video coding (H.265/MPEG-HEVC) using six different compression rates. The degraded images were presented in rapid alternation with the original images. In this setting, the presence of SSVEPs is a neural marker that objectively indicates the neural processing of the quality changes that are induced by the video coding. We tested two different machine learning methods to classify such potentials based on the modulation of the brain rhythm and on time-locked components, respectively. Main results. Results show high accuracies in classification of the neural signal over the threshold of the perception of the quality changes. Accuracies significantly correlate with the mean opinion scores given by the participants in the standardized degradation category rating quality assessment of the same group of images. Significance. The results show that neural assessment of video quality based on SSVEPs is a viable complement of the behavioral one and a significantly fast alternative to methods based on the P3 component.BMBF, 01GQ0850, Bernstein Fokus Neurotechnologie - Nichtinvasive Neurotechnologie für Mensch-Maschine Interaktio

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.EC/FP7/611570/EU/Symbiotic Mind Computer Interaction for Information Seeking/MindSeeEC/FP7/625991/EU/Hyperscanning 2.0 Analyses of Multimodal Neuroimaging Data: Concept, Methods and Applications/HYPERSCANNING 2.0DFG, 103586207, GRK 1589: Verarbeitung sensorischer Informationen in neuronalen Systeme

Ausweitung der Grenzen der Brain-Computer-Interface-Technology : Herausforderungen in neuartigen Anwendungen und großangelegten Studien

Since the first studies in the 70s, Brain-Computer Interface (BCI) research has had an exponential development thanks to the high potential in improving life of severely impaired people. In the last years, the applicability of BCI extended also to non-medical applications as a complementary tool to provide interesting insights into the brain processes correlated with behavior. One of the most exploited measurement techniques used in BCI is Electroencephalography (EEG), because of its portability, contained costs and direct recording of neural activity. This thesis focuses on EEG-based BCI, with the aim of investigating cutting-edge applications, novel designs and the possibility of broadening this technology on large scale. According to the specific application, different brain features were analyzed, from visual to higher cognitive event-related potentials (ERPs), from the modulation of spontaneous oscillatory brain rhythms to that of the sensorimotor rhythm (SMR). Signal processing and machine learning algorithms were tailored to extract representative features and predict the user’s state or the user’s intention for the BCI operation. This thesis extends the boundaries of state-of-art BCI applications from several points of view. Firstly, the feasibility of a steady-state visual evoked potential (SSVEP)-based paradigm for video quality assessment was proven, which led to the collection of informative neural features in a much faster way than previous ERP-based paradigms. Secondly, two novel ERP-BCIs for mental typewriting independent of gaze-shifts were developed, which could be operated online with high performance by healthy users and could represent a valid alternative for patients with severely impaired oculomotor control. Thirdly, a revised version of the standard BCI design was investigated for the study of subjective relevance in the scope of information retrieval (IR) systems, bringing new insight into brain processes that can enhance the interaction between man and machine. Lastly, the widely employed SMR-BCI paradigm was applied on large scale and in a fully automatic way. The aim was to test how co-adaptive machine learning algorithms, successful when user-tailored, would perform in a realistic scenario without any manipulation from BCI-experts. The overall work shows the versatility of BCI as a successful supplement of the existing technology and as a tool to improve the lives of people, also pointing out the main limitations with critical view to trigger new research developments.Seit den ersten Studien in den 1970er Jahren, erlangte die Forschung in Gehirn-Computer Schnittstellen (engl. Brain-Computer Interface BCI) eine exponentielle Entwicklung dank der hohen Wahrscheinlichkeit der Lebensverbesserung von stark beeinträchtigten Personen. In den vergangenen Jahren hat sich die Verwendung von BCI auch auf nicht medizinische Anwendungen als ein ergänzendes Werkzeug ausgebreitet, welches interessante Einblicke in die Korrelation von Hirn-Prozessen und Verhalten gibt. Eine der in BCI meist verwendeten Messungstechniken ist die Elektroenzephalographie (EEG), aufgrund ihrer Mobilität, geringen Kosten und direkter Aufzeichnung von neuraler Aktivität. Diese Arbeit befasst sich mit dem EEG-basierten BCI. Ziel ist die Forschung an innovativen Anwendungen, neuen Designs und der Möglichkeit diese Technologie in großem Ausmaß zu verbreiten. Je nach spezifischer Anwendung wurden folgende verschiedene Hirnströme analysiert: Visuelle und kognitive Ereigniskorrelierte Potentiale (eng. event-related potentials (ERPs)), Modulation von spontan schwankenden Hirnsignalen und sensomotorischen Rhythmus (SMR). Algorithmen aus der Verarbeitung und dem Maschinellen Lernen wurden zugeschnitten um repräsentative Hirnströme zu extrahieren und den Zustand und die Absichten des Nutzers vorherzusagen. Diese Arbeit erweitert den aktuellen Stand der Technik der BCI-Anwendungen in vieler Hinsicht. Erstens, wurde die Durchführbarkeit eines BCI Paradigmas zur Bewertung von Video Qualität bewiesen. Dies ermöglicht eine wesentlich schnellere Sammlung von informativen Hirnströmen als bei bisherigen ERP-basierten Paradigmen. Zweitens, wurden zwei neue Blick-unabhängige ERP-BCIs zur mentalen Texteingabe entwickelt. Diese konnten online mit hoher Präzision von gesunden Nutzern durchgeführt werden. Diese Paradigmen stellen eine Alternative für Patienten mit einer hohen Beeinträchtigung der Augenbewegung an. Drittens, wurde eine veränderte Version des Standard BCI Designs zur Untersuchung von subjektiver Relevanz im Bereich der Systeme zur Informationsgewinnung erforscht. Dies brachte neue Einsicht in Hirnprozesse welche die Interaktion zwischen Mensch und Maschine fördern kann. Zuletzt wurde das sehr bekannte SMR-BCI Paradigma auf eine Großzahl von Nutzern voll automatisch angewandt. Das Ziel war zu testen, welche Ergebnisse co-adaptive maschinell lernende Algorithmen in einem realistischem Szenario ohne jegliche Manipulation von BCI-Experten erzielen würden. Die Arbeit zeigt im allgemein die Vielseitigkeit des BCI als eine erfolgreiche Ergänzung zu den bereits existierenden Technologien und Werkzeugen welche der Verbesserung des Lebens der Menschen dienen. Die Arbeit deutet jedoch auch kritisch auf die Einschränkungen hin, um neue Forschungsentwicklungen auszulösen

Assessing the Depth of Cognitive Processing as the Basis for Potential User-State Adaptation

Objective: Decoding neurocognitive processes on a single-trial basis with Brain-Computer Interface (BCI) techniques can reveal the user's internal interpretation of the current situation. Such information can potentially be exploited to make devices and interfaces more user aware. In this line of research, we took a further step by studying neural correlates of different levels of cognitive processes and developing a method that allows to quantify how deeply presented information is processed in the brain.Methods/Approach: Seventeen participants took part in an EEG study in which we evaluated different levels of cognitive processing (no processing, shallow, and deep processing) within three distinct domains (memory, language, and visual imagination). Our investigations showed gradual differences in the amplitudes of event-related potentials (ERPs) and in the extend and duration of event-related desynchronization (ERD) which both correlate with task difficulty. We performed multi-modal classification to map the measured correlates of neurocognitive processing to the corresponding level of processing.Results: Successful classification of the neural components was achieved, which reflects the level of cognitive processing performed by the participants. The results show performances above chance level for each participant and a mean performance of 70–90% for all conditions and classification pairs.Significance: The successful estimation of the level of cognition on a single-trial basis supports the feasibility of user-state adaptation based on ongoing neural activity. There is a variety of potential use cases such as: a user-friendly adaptive design of an interface or the development of assistance systems in safety critical workplaces

Large-Scale Assessment of a Fully Automatic Co-Adaptive Motor Imagery-Based Brain Computer Interface.

In the last years Brain Computer Interface (BCI) technology has benefited from the development of sophisticated machine leaning methods that let the user operate the BCI after a few trials of calibration. One remarkable example is the recent development of co-adaptive techniques that proved to extend the use of BCIs also to people not able to achieve successful control with the standard BCI procedure. Especially for BCIs based on the modulation of the Sensorimotor Rhythm (SMR) these improvements are essential, since a not negligible percentage of users is unable to operate SMR-BCIs efficiently. In this study we evaluated for the first time a fully automatic co-adaptive BCI system on a large scale. A pool of 168 participants naive to BCIs operated the co-adaptive SMR-BCI in one single session. Different psychological interventions were performed prior the BCI session in order to investigate how motor coordination training and relaxation could influence BCI performance. A neurophysiological indicator based on the Power Spectral Density (PSD) was extracted by the recording of few minutes of resting state brain activity and tested as predictor of BCI performances. Results show that high accuracies in operating the BCI could be reached by the majority of the participants before the end of the session. BCI performances could be significantly predicted by the neurophysiological indicator, consolidating the validity of the model previously developed. Anyway, we still found about 22% of users with performance significantly lower than the threshold of efficient BCI control at the end of the session. Being the inter-subject variability still the major problem of BCI technology, we pointed out crucial issues for those who did not achieve sufficient control. Finally, we propose valid developments to move a step forward to the applicability of the promising co-adaptive methods

Predictor of BCI performance.

<p>The predictor is calculated from 2.5 minutes of recording of the brain in resting state with eyes open and correlated with the mean feedback accuracy for each participant (blue dots). The back dashed line pictures the linear regression between the predictors and the accuracies (r = 0.53). The magenta dots are the values detected as outliers. After the exclusion of the outliers, a higher correlation of 0.66 is reached (magenta line).</p

Grand average ERD/ERS for class combination. <i>left-right</i> and intervention groups in Berlin.

<p>‘N’ is the number of participants of each group. From left to right: runs 1–3, runs 4–5, runs 6–7. The time plots in the first rows picture the evolution of the ERD/ERS for about 6000 ms at C3 (thick lines) and C4 (thin lines). At time 0 is the onset of the cue, at times 1000–4000 the display of the feedback. Magenta lines refer to <i>left</i> MI trials, green lines to <i>right</i> MI trials. The scalp plots underneath refer to the shaded areas of the time plots and show the distribution of the ERD/ERS. In the second rows, the scalp plots of the <i>left</i> MI trials, in the third rows the scalp plots of the <i>right</i> MI trials and in the fourth the scalp plots of the sign-r².</p